1. Field of the Invention
The present invention relates to event stream processing and, more particularly, to systems and methods for scalable real time processing of a substantial volume of event data streams.
2. Description of the Related Art
Relational database management systems have become an essential part of the U.S. and world economy. Once historical data has been stored in the database, it can be queried in various ways, to provide for decision support. While traditional relational database management systems have many advantages, some challenges still remain.
Modern enterprises now generate an enormous number of events, and an enormous amount of corresponding event data. For example, although e-commerce and Radio-Frequency Identification tags have potential to automate transactions and help in supply chain management, large scale adoption of e-commerce services and RFID services may generate an overwhelming number of events, and an overwhelming amount of event data.
In addition to such difficulties, some sectors of services additionally require real time processing, or near real time processing, of event data streams. Financial services events, for example events related to the buying and selling of positions in stock markets, may need to process streams of data for buy/sell price events quickly, to support financial transactions. Furthermore, it should be understood that such real time (or near real time) processing may be highly desirable, or may provide very compelling business advantages, for many kinds of services.
Recent advances in healthcare services can rapidly generate important events (and corresponding important event data) in automated patient monitoring or testing, e-prescriptions, and electronic medical records. Additional event examples are security services events, which include, but are not limited to, services events for access control, authentication, encryption, non-repudiation and public key infrastructure. Further examples of services events, such as collaboration services events include, but are not limited to, services events for social networking, virtual workgroups, instant messaging, surveys, polls, discussion boards and document sharing.
Such services, and their potential for corresponding number of generated events and amount of event data, can grow very rapidly. For example, it is estimated that, beginning in December of 2008, a very popular social networking service exceeded 130 million monthly active user mark, and shortly thereafter appeared to increase significantly at a rate of 600,000 or perhaps even 700,000 new users each day.
Furthermore, powered by recent advances in mobile communications technology, the variety of different ways of delivering services is increasing rapidly. In the past, people were constrained to consuming such services in their homes or offices. Today, people may wish to consume any of the foregoing services—at any time and in any place, as mobile communication services (potentially generating rapidly increasing numbers of events and amounts of event data).
Adequate processing headroom is needed to accommodate such increases in processing load. As a general matter, scalability indicates an ability either: to handle growing amounts of work in a graceful manner with adequate headroom; or to be readily enlarged, so as to increase total throughput under an increased load, when resources (typically hardware) are added. For example, a system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system. However, if the design fails with increased load, despite increased resources, then it does not scale.
Accordingly, it should be understood that innovations are needed to handle real time, or near real time, processing of substantial and increasing volume of event data streams, in a scalable and efficient manner.